Secrets Rotation Framework

Every secret should be treated as temporary. A rotation framework defines how secrets are stored, delivered, replaced, and revoked so that a leak has a bounded impact and replacement is routine rather than an emergency.

The strongest rotation strategy is to remove the secret entirely. Workload identity, federated tokens, and IAM roles let services authenticate without stored values, so apply rotation mainly to what remains: third-party keys, legacy database accounts, signing keys, and certificates.

How it works

A secrets manager holds versioned values and exposes them through an API with fine-grained access control. Workloads authenticate with their platform identity and fetch secrets at start-up or on a refresh interval. Rotation uses a two-phase pattern: create a new version, make the backend accept both, update consumers, verify, and then retire the old version. For databases, dynamic credentials issue a unique user per lease that expires automatically.

Implementation steps

1. Inventory secrets by type, owner, location, and current age, using scanners on repositories and CI.
2. Classify by blast radius and set maximum lifetimes, for example 30 days for database passwords and 90 days for third-party API keys.
3. Move secrets into the managed store and remove them from code, images, and CI variables.
4. Make applications reload credentials without restart, or use sidecar agents that renew files.
5. Implement rotation functions with a validation step before retiring the old version.
6. Alert on secrets nearing expiry and on failed rotations.
7. Write and rehearse a break-glass revocation runbook.

Start with the highest-risk secrets, such as production database administrators and cloud root credentials, then expand outward. Early success on a few critical items builds support for the wider program.

Common pitfalls

• Rotating a secret in the store but not in the consuming system, breaking authentication.
• Deleting the old version before every consumer has switched.
• Caching secrets in memory forever.
• Forgetting certificates and signing keys, which expire quietly.

Metrics to watch

Maximum and median secret age, percentage of secrets rotated automatically, rotation failure rate, time to revoke a compromised secret, and the number of secrets found by repository scanners.

Pair rotation metrics with detection signals such as access anomalies from the secrets manager audit log. Rotation reduces the window of exposure, while monitoring tells you when a window was actually used.

Example rotation policy

secret: orders-db-password
owner: orders-team
store: managed-secrets
rotation:
  interval_days: 30
  strategy: dual-user
  steps:
    - create_new_credential
    - grant_same_privileges
    - publish_new_version
    - verify_connections_for_minutes: 15
    - revoke_previous_credential
alerts:
  expiry_warning_days: 7
  on_failure: page-oncall

Key takeaways

• Inventory first, then set lifetimes by blast radius.
• Overlap old and new versions to rotate without downtime.
• Applications must tolerate reloads.
• Rehearse emergency revocation before you need it.