Training Pipeline Produces Corrupted Model Artifact

A corrupted training dataset has already produced a technically valid model artifact — it trains successfully, passes basic health checks, and could be deployed without a single infrastructure error occurring anywhere — and that artifact is already serving live production traffic. Trainer throughput and Deployment Pipeline uptime remain fully healthy throughout: this does not look like, and must not be mistaken for, an infrastructure failure. Five business invariants govern every decision here: (1) artifact promotion must require passing a real trust/validation gate before reaching Deployment Pipeline, not merely a shallow health check; (2) a model already serving must be rollback-able to a known-good prior artifact; (3) lineage — which dataset produced which artifact, which artifact is currently serving — must remain fully auditable at all times, even mid-incident; (4) corrupted artifacts must never be silently deleted or overwritten, they must be explicitly quarantined/marked distrusted instead; (5) business decisions already made using the corrupted model's output before detection are a real, permanent, non-resetting historical exposure. Decide whether to roll back immediately to a last-known-good artifact or hold promotion open for a full retrain-and-revalidation cycle, whether to harden the promotion gate so the same mistake cannot recur, and how to keep the incident's forensic record complete without ever destroying it.

Difficulty

Principal

Expected Time

~25 min

Category

Data Pipelines