5 MLOps Mistakes I Keep Seeing in Production (And How to Fix Them)

Manicandane PARI

— 2 min read
Share

Most MLOps content focuses on tools—trackers, containers, orchestration. Useful, but not where teams lose the most time. The real pain comes from skipping “optional” practices that become mandatory once the model is live.

1) Shipping a model with no monitoring

Models don’t fail like software. APIs throw 500s. Models degrade quietly and keep returning plausible wrong answers.

What to monitor:

  • Tabular: feature distributions (means/quantiles, categorical frequencies). Calibrate thresholds to natural variance and seasonality (day-of-week, campaigns), not arbitrary rules.
  • Data quality (a common production incident source): schema/range violations, missingness spikes, cardinality explosions, feature freshness.
  • Non-tabular (text/images/embeddings): representation-level shifts (distance/similarity distribution changes) and output distribution shifts. Track model score distributions only if the scoring function is stable and (when applicable) calibrated.
  • Business KPIs: always tie model version → CTR/conversion/retention. A KPI drop correlated with a deploy is often more actionable than any drift statistic.

Minimum viable: a daily job logging key outputs + data quality checks + anomaly alerts. Day one.
Define ownership: who gets paged, what triggers a rollback, and what “good” looks like for each KPI.

2) Calling “versioning” a .pkl in S3

If you can’t reconstruct exactly what produced the model, rollback becomes archaeology.

A versioned model answers:

  • Which code (Git SHA)
  • Which data (hash / versioned pointer)
  • Which config (resolved hyperparams)
  • Which metrics (training-time evals)
  • Which runtime (deps / Docker). Serialized artifacts can break or change semantics across library versions—treat the runtime as part of the version.

Common failure: preprocessing drifts separately (scalers/encoders/feature logic). Ensure a single source of truth for transforms and fitted params—packaged together when possible (e.g., sklearn.Pipeline).

Rule: if stored metadata can’t reproduce the same artifact, it’s not versioned—it’s just saved.

3) Running production training from a notebook

Notebooks are great for EDA. They’re fragile as scheduled/CI training pipelines.

  • Diffs are unreadable (JSON + metadata noise)
  • Config becomes messy (global state, dev/stage/prod hacks)
  • Testing is near-zero (logic in cells never gets unit-tested)
  • Build artifacts are unclear (hard to package, pin deps, and promote through environments like software)

Fix without a rewrite: move core logic to modules + train.py, add a config layer (Hydra or YAML+dataclasses), keep notebooks for exploration and reporting.

4) Deploying without staging validation

“94% on the test set” ≠ “works in production.”

Staging should mirror prod (runtime, hardware class, feature pipeline) and use realistic data (anonymized samples or replay). Validate latency under load (p95/p99), memory footprint, and throughput. This catches boundary failures: schema mismatches, null patterns, concurrency issues, and postprocessing edge cases.

Promotion basics:

  • Shadow: run the candidate on real traffic, log outputs, don’t serve results.
  • Canary: start small (5–10%), monitor, then ramp. Rollback is cheap if you designed for it (routing, flags, kill switch, runbook).

5) Letting ML technical debt compound

This doesn’t explode. It slows you down until every change feels risky.

  • Training/serving skew (slightly different feature logic)
  • Hardcoded “fit” constants (means/std/thresholds that go stale)
  • No pipeline entry validation (bad schema/data silently trains a worse model)

Fix: enforce a single source of truth for transforms, fit+persist parameters during training, and add a lightweight schema/quality gate (pandera or Great Expectations).

The thread

None of this requires exotic tools. It’s cheap early and expensive later. The mindset shift is simple: a model isn’t a deliverable—it’s a component you’ll operate, monitor, and debug for years.

Read more