Secret Leakage Containment Incident

A credential used by Applications and External Integrations has been found in plaintext in a Logging record and in a CI/CD build artifact — it must now be treated as believed-exposed. The system is otherwise completely healthy: there is no outage, no scaling problem, and no capacity pressure anywhere in this incident. The only problem is risk, and risk does not show up on an availability dashboard. Four business invariants govern every decision here: (1) a believed-exposed credential's blast radius must be understood — which systems and integrations actually use it — before broad revocation, but investigation must not become an excuse to delay containment indefinitely; (2) revoking a credential that is still relied on by a critical, not-yet-migrated integration, without a replacement in place, creates a real, avoidable outage — this must be sequenced, not instant; (3) every use of the exposed credential after exposure was identified must be recorded as an auditable historical fact, whether or not it was ultimately malicious; (4) containment — narrowing scope, rotating to a new credential, monitoring for misuse — is preferred over broad, simultaneous revocation without a sequenced replacement plan. This incident's central security invariant sits above all four: once a credential is believed exposed, continuing to treat it as fully trusted, taking no containment action at all, is never acceptable, regardless of how disruptive rotation might be to dependent systems. Doing nothing looks completely safe on every availability metric while a real, permanent, non-resetting count of suspected exposed-credential uses accrues in the background. The opposite shortcut — revoking the credential broadly and immediately, before any dependent-system cohort has migrated to a replacement — fixes the security exposure instantly but breaks every not-yet-migrated integration at once, a real and entirely avoidable outage. Decide how to narrow trust and monitor for misuse while blast radius is confirmed, how to sequence cohort-by-cohort migration to a new credential before fully revoking the old one, and what historical review is required before this incident can honestly be called closed.

Difficulty

Principal

Expected Time

~20 min

Category

Security & Identity